Datamart Load

Data that is processed on a recurring basis needs to be loaded into the datamart so that clients can access and use it. The loading process can be performed manually through various methods such as the Aunsight UI, toolbelt, or API, or it can be integrated into a workflow for automation.

Multiple tables can be loaded simultaneously, allowing for efficient data loading. Throughout the loading process, the task status and logs provide progress updates, ensuring transparency and visibility into the loading operations.

The tables are loaded from Atlas datasets, which can be in DSV (Delimited Separated Values). These formats offer flexibility and compatibility with different data structures.

Loading supports different write modes, including append and overwrite. The append mode adds new data to existing tables, while the overwrite mode replaces the existing tables with the new data. This allows for flexibility in managing the data loading process.

It's important to note that views cannot be loaded since they are essentially queries based on other existing tables. Views serve as a convenient way to access and analyze data already present in the datamart.

As adoption and usage increase, there are plans to introduce additional write modes to further enhance the loading capabilities and accommodate various data loading scenarios.

Instructions for filling out the JSON object

Tables:

  • Within the tables array, each object represents a table to load.
  • Provide the id for each table to load in the id field.
  • Specify the dataset from which to load the table in the dataset field.
  • If there are specific loading options for the table, you can include them in the load_options field as a JSON object. Otherwise, you can leave it empty {}.
  • Assign the record field with the Atlas record ID from which to load the table.
  • Set the write_mode field to either append or overwrite to determine the mode to write with. Choose append to add data to an existing table, or overwrite to replace the existing table with the new data.
{
  "tables": [
    {
      "id": "table1",
      "dataset": "dataset1",
      "load_options": {},
      "record": "record1",
      "write_mode": "append"
    },
    {
      "id": "table2",
      "dataset": "dataset2",
      "load_options": {},
      "record": "record2",
      "write_mode": "overwrite"
    }
  ]
}